Head mounted device (HMD) system having interface with mobile computing device for rendering virtual reality content

ABSTRACT

A head mounted display (HMD) system is provided. The system includes a housing and optics disposed in the housing. The system further includes a processor disposed in the housing and shutter control logic interfaced with the processor. The shutter control logic is configured to operate to alternate between opening a left eye-side of the optics and closing a right eye-side of the optics to opening the right eye-side of the optics and closing the left eye-side of the optics. A holder portion is integrated with the housing. The holder portion is configured to receive a smart device having a display screen. The display screen is oriented toward the optics of the housing in a direction that is toward a face of a user when the HMD system is worn by the user. The shutter control in operation is configured to enable a three-dimensional view of image date rendered on the display screen of the smart device when viewed through the optics of the housing.

CLAIM OF PRIORITY

This application is a nonprovisional of U.S. Provisional PatentApplication No. 62/006,022, filed on May 30, 2015, and which is hereinincorporated by reference.

BACKGROUND

Field of the Invention

The present invention relates to headsets used for viewing media contentand more particularly, headsets that can integrate smart devices andprocess image data for 3D rendering.

Description of the Related Art

The computing industry and the video game industry have seen manychanges over the years. As computing power has expanded, developers ofvideo games have created game software that have adapted to theincreased computing power. To this end, video game developers have beencoding games that incorporate sophisticated operations and mathematicsto produce a very realistic game experience.

These games are presented as part of a gaming system including gameconsoles, portable game devices, and/or provided as services over aserver or the cloud. As is well known, the game console is designed toconnect to a monitor (usually a television) and enable user interactionthrough handheld controllers/input devices. A game console may includespecialized processing hardware, including a CPU, a graphics processorfor processing intensive graphics operations, a vector unit forperforming geometric transformations, and other glue hardware, firmware,and software. The game console may be further designed with an opticaldisc tray for receiving game compact discs for local play through thegame console. Online and multi-player gaming is also possible, where auser can interactively play against or with other users over theInternet. As game complexity continues to intrigue players, game andhardware manufacturers have continued to innovate to enable additionaland more realistic interactivity.

A growing trend in the computer gaming industry is to develop games thatincrease the interaction between the user and the gaming system. One wayof accomplishing a richer interactive experience is to use wireless gamecontrollers whose movement and gestures are tracked by the gamingsystem. These movements and gestures are used as inputs for the game.Gesture inputs, generally speaking, refer to having an electronic devicesuch as a computing system, video game console, smart appliance, etc.,react to some gesture made by the user while playing the game that arecaptured by the electronic device.

Another way of accomplishing a more immersive interactive experience isto use a head-mounted display (HMD). A head-mounted display is worn bythe user and can be configured to present various graphics, such as aview of a virtual space, in a display portion of the HMD. The graphicspresented on a head-mounted display can cover a large portion or evenall of a user's field of view. Hence, a head-mounted display can providean immersive experience to the user. As connectivity to the internetcontinues to increase, more configurations of HMD systems have beenintroduced, and each with some advantages and associated limitations.

It is in this context that embodiments of the invention arise.

SUMMARY

Embodiments are described regarding systems, methods and apparatus thatenable rendering of content to a display screen, and in particular on ahead mounted display (HMD). The content rendered may include rich threedimensional content, which may be rendered in the context of immersivevirtual reality content.

In one embodiment, a device is provided that includes a slot or holderfor receiving a smartphone. The device is configured with straps orhead-attach structure so that a smartphone can be located in the slot orholder. The screen of the smartphone, in one configuration, is viewableto the user much in the same way a traditional HMD has a screenintegrated therein. The device, therefore, does not include the displayscreen, but instead the display screen of the smartphone is used torender images to be viewed by the user. In one implementation, thesmartphone will communicate with the Internet to access content oraccess content from storage of the smartphone. In a connected mode, thesmartphone may obtain a streaming content or downloaded content, whichcan be rendered on the smartphone when the smartphone is operating as ahead mounted device. In one embodiment, the content rendered on thedisplay of the phone is distorted to produce three dimensional images,and the optics of the smartphone holding device can un-distort theimages so that the content displayed by the smartphone, when viewed viathe optics of the device, will appear to be rich three dimensionalimage/video/interactive data. In one configuration, the head mounteddevice will include shutter control logic that will allow control of theright and left eye optics of the device, such that the image data israpidly shuttered to show left, right, left, right, left, right . . .and so on, images, which in essence enable the image data rendered bythe head mounted display (i.e., holding device and smartphone) to bethree-dimensional, rich and immersive.

A head mounted display (HMD) system is provided. The system includes ahousing and optics disposed in the housing. The system further includesa processor disposed in the housing and shutter control logic interfacedwith the processor. The shutter control logic is configured to operateto alternate between opening a left eye-side of the optics and closing aright eye-side of the optics to opening the right eye-side of the opticsand closing the left eye-side of the optics. A holder portion isintegrated with the housing. The holder portion is configured to receivea smart device having a display screen. The display screen is orientedtoward the optics of the housing in a direction that is toward a face ofa user when the HMD system is worn by the user. The shutter control inoperation is configured to enable a three-dimensional view of image daterendered on the display screen of the smart device when viewed throughthe optics of the housing.

In another embodiment, a system for rendering virtual reality content isprovided. The system includes a housing to be attached to a head of auser via a strap. The system further includes optics integrated in thehousing. The display optics are oriented on an internal side of thehousing that is configured for orientation toward eyes of the user whenthe housing is attached to the head of the user. A holder is integratedin the housing. The holder is configured to receive a smartphone orsmart device (i.e., tablet or device with a screen). In one example, ascreen of the smartphone is oriented toward the display optics of thehousing. The smartphone has a data connection with the holder to enabledata communication between electronics of the housing and electronics ofthe smartphone. Shutter control logic is interfaced with the electronicsof the housing. Optics control logic is interfaced with the optics. Aprocessor is configured to interface with the shutter control logic andthe optics control logic. The processor is configured to set the shuttercontrol logic to successively switch between rendering image dataobtained from the screen of the smartphone to only a left eye-side ofthe optics and then only to a right eye-side of the optics. Theprocessor is further configured to interface with the optics controllogic so that rendering is optimized for when the image data is providedto the left and right eye-sides of the optics. The shutter control logicenables display of three-dimensional images, as perceived by a user,from the image data rendered on the screen of the smartphone when viewedthrough the optics of the housing.

In some implementation, the system includes a device interface (wired orwireless) for connecting the smartphone to the housing.

In some implementation, the system includes an inertial sensor of thehousing. The inertial sensor provides movement and orientation data ofthe housing, and the movement and orientation data of the housing iscommunicated to the smartphone. The smartphone uses the movement andorientation data of the housing along with movement and orientationproduced by an inertial sensor of the smartphone to fine-tune movementand orientation of the housing to influence or update a view directionand perspective into a virtual reality scene rendered by the smartphone.

In some implementation, the system includes combining at least some datafrom the inertial sensor of the housing with data from the inertialsensor of the smartphone to improve tracking of the housing.

In some implementation, the system includes a plurality of lightemitting diodes disposed on a surface of the housing, the light emittingdiodes used for tracking position of the housing using at least onecamera.

In some implementation, the system includes a plurality of photo diodesdisposed on a surface of the housing, the photo diodes used for trackingposition of the housing using at least one light emitting devicepositioned in space where the housing is to be tracked.

In some implementation, the system includes circuitry of the electronicsof the housing for processing a see-through mode, wherein a front facingcamera of the smartphone is activated. The shutter control logic ispaused and the optics control logic enables display of image data fromthe screen of the smartphone that shows a real-world view via the frontfacing camera of the smartphone, when in the see-through mode.

In some implementation, the system includes circuitry of the electronicsof the housing for processing a see-through mode, wherein a front facingcamera of the housing is activated. The shutter control logic is pausedand the optics control logic enables display of image data from thescreen of the smartphone that shows a real-world view via the frontfacing camera of the housing, when in the see-through mode.

In some implementation, the system includes a network interfaceintegrated with the housing. The network interface providing access tothe network for connecting to an internet site having digital content.The digital content is one of streaming content or interactive streamingcontent, and wherein the streaming interactive content is to be renderedby the smartphone for display via the optics of the housing.

In some implementation, the system includes a network interface providedby the smartphone. The network interface provides access to the networkfor connecting to an internet site having digital content. The digitalcontent is one of streaming content or interactive streaming content,wherein the streaming interactive content is to be rendered by thesmartphone for display via the optics of the housing.

In some embodiments, the optics that are integrated into the housinginclude a left eye optic and a right eye optic.

In some embodiments, the housing includes a device input (i.e.,interface for communication with input devices or information. Thedevice input is configured for receiving user input for controlling atleast part of an interactive content scene rendered in the image datagenerated by the smartphone.

In one embodiment, the user input is via a controller, a hand gesture, atouch gesture, or voice control, or a combination thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a user wearing a device, e.g., housing that canreceive a smartphone or smart device, the device is in a form of glassesor a head mounted device (HMD), in accordance with one embodiment.

FIG. 1B illustrates a side view of the glasses, which includeelectronics and a holder for a smartphone, in accordance with oneembodiment.

FIG. 1C illustrates a perspective view of the glasses with a slot forholding the smartphone, in accordance with one embodiment.

FIG. 1D illustrates an example smartphone, which can also include asmart device, a tablet device or any device having a screen fordisplaying content, in accordance with one embodiment.

FIG. 2 illustrates a user wearing the HMD device, which includes thesmartphone and interaction with a scene rendered by the HMD device, inaccordance with one embodiment.

FIG. 3A illustrates an example form-factor of the housing, e.g.,receiver unit that can hold a smartphone, wherein the camera of thesmartphone is front facing,

FIG. 3B illustrates an example configuration wherein the smartphone isconnected by a wire to the housing receiving unit, and optics of thereceiving unit/housing, in accordance with one embodiment.

FIG. 4A illustrates example electronics that are part of the receiverunit or housing, and the interface of the electronics with the opticsand the smartphone, in accordance with one embodiment.

FIG. 4B illustrates an example where software of the smartphone isconfigured to process the frame selection and left/right eye selection,including up-rendering (scale-up), and processing for display andviewing through the optics of the HMD housing, in accordance with oneembodiment.

FIG. 4C illustrates an example wherein the shutter logic sets the opticsto alternating left/right viewing, in accordance with one embodiment.

FIG. 5 illustrates an example of electrics of a housing, which caninclude the shutter control logic, the controller optics, a processor,network interface, inertial sensor(s), memory, frame buffers, and deviceinterfaces for connecting with the smartphone, and internet access todigital content, in accordance with one embodiment.

FIG. 6 illustrates an example where co-located users can share contentlocally, e.g., peer-to-peer, including with a cloud system, inaccordance with one embodiment.

FIG. 7 illustrates another example of an HMD system, which uses asmartphone and shutter logic to provide rich 3D content, in a VRexperience.

FIG. 8A illustrates an example of a user wearing an HMD, interfacingwith a computing device and a camera used for tracking, in accordancewith one embodiment.

FIG. 8B illustrates a plurality of LEDs that can be used for cameratracking, and in some cases, photo diodes can be integrated onto thesurface of the housing to provide tracking using a laser or lightemitting source(s), in accordance with one embodiment.

FIG. 9 illustrates one example of gameplay using the client system.

FIG. 10 illustrates an example of an HMD, tracking and play via a gameconsole, in accordance with one embodiment.

FIG. 11 illustrates example electronics that can be made part of thehousing or HMD system, in accordance with one embodiment.

FIG. 12 provides an example of a datacenter and access to interface withmultiple users, e.g., via a network for multi-player interaction,communication and/or play, in accordance with one embodiment.

DETAILED DESCRIPTION

Embodiments are described regarding systems, methods and apparatus thatenable rendering of content to a display screen. The content renderedmay include rich three dimensional content, which may be rendered in thecontext of immersive virtual reality content. The content may be gamerelated, entertainment related, communication related, socialcommunication or sharing related, or combinations of these and othertypes of media and information. In some examples described withreference to FIGS. 1A-7, head set devices are shown that may beconfigured to communication with a second computing device. In oneexample, the second computing device can be a smartphone, tablet,computer, watch, or some device that can communicate to obtain imagecontent and enable rendering of such image content.

In one implementation, a head mounted device is provided, which includesa slot or holder for receiving a smartphone, so as to allow the screenof the smartphone to be the display of the head mounted device. Thus, inthis example, the head mounted device does not include the displayscreen, but instead will use the display screen of the smartphone, wheninserted or placed in a holder of the head mounted device. Thesmartphone may be connected to the head mounted device via wired orwireless connection. In such an implementation, the smartphone willcommunicate with the Internet to access content, such as streamingcontent or downloaded content, which can be rendered on the smartphonewhen the smartphone is inserted or coupled to the head mounted device.In a basic form, the head mounted device will include electronics forcommunicating with the smartphone, either wired or wireless. The headmounted device may include optics for enabling viewing of the contentrendered on the display.

In one embodiment, the content rendered on the display of the phone isdistorted to produce three dimensional images, and the optics of thehead mounted display can un-distort the images so that the contentdisplayed by the smartphone, when viewed via the optics of the headmounted device, will appear to be rich three dimensionalimage/video/interactive data. In another embodiment, the contentrendered on the display of the smartphone, when inserted into the headmounted device, will render images that can be processed by the headmounted device to appear as three-dimensional. In one embodiment, thehead mounted device will include shutter control logic that will allowcontrol of the right and left eye optics of the head mounted device,such that the image data is rapidly shuttered to show left, right, left,right, left, right . . . and so on, images, which in essence enable theimage data rendered by the head mounted display to be three-dimensional,rich and immersive.

FIG. 1A illustrates an example of a user wearing a head mounted device,which may include a smartphone 60 held, inserted, clipped, slid into,associated to, or paired to the head mounted device. The glasses 50 mayin one embodiment define the HMD, when the smartphone 60 is insertedtherein. The head mounted device can be a simple shell device that holdsthe smartphone 60 in place, and provides optics for viewing through tothe screen of the smartphone 60. In another embodiment, the head mounteddevice can be more intelligent, which can include electronics forcommunicating with the smartphone 60 when connected, in proximity, orassociated to the head mounted device. In this illustration, the headmounted device is shown as glasses 50, which have a smartphone connectedor clipped to the glasses.

The smartphone can use its native wireless communication circuitry tocommunicate with the network 110, such as the Internet. The smartphone60 can communicate to various content sites including cloud gamingcontent, cloud entertainment content, social media content, or any othertype of content that is accessible over the Internet or privatenetworks. In one embodiment, the content being accessed by thesmartphone 60 can be downloaded and executed by the smartphone once thecontent has been received. In another embodiment, the content can bestreamed from the Internet source, and delivered to the smartphone 60when needed. In one embodiment, the content is streamed from a serviceprovider that provides games, content, entertainment content, or othermultimedia for consumption by the smartphone when used with a headmounted device or glasses 50.

FIG. 1B illustrates a side view of a pair of glasses 50 that includeelectronics for interfacing with a smartphone 60 when the smartphone 60has been inserted or held by the glasses 50. In this example, thesmartphone 60 sits vertically in a slot or holder of the glasses 50,such that the smartphone 60 camera faces forward to allow a see-throughor pass-through view of the environment when the glasses 50 are worn bya user. The body of the glasses 50 defines the housing 70. The housing70 can hold electronics 90 and can include a holder 91. The holder, asdescribed herein, shall include any structure for holding, clipping,clamping, securing, attaching, or otherwise maintaining the smartphonein place. In one embodiment, activation of the see-through mode (i.e.,transparent or semi-transparent mode) is used when the user isinteracting with a virtual scene displayed by the smartphone 60 devicethrough the glasses 50, and the user wishes to disconnect from thevirtual scene to interact with another person, take a phone call, pausegame, pause session, or an interactive session or communication. Thesee-through mode can also be automatically triggered, such as whensignals are received or when safety alerts or notifications are needed.

FIG. 1C illustrates an example of the glasses, which include electronics90 for communicating with a plug, Bluetooth, Wi-Fi, and other wirelessprotocol, or wired connection, plug, connector, of the glasses 50 andthe smartphone 60 when connected together. In this example, thesmartphone screen 60 a side will face the location of the user's eyes orgenerally toward the eyes when the glasses 50 are worn or inserted overthe face of the user. The rear side of the smartphone can include asmartphone camera which can face forward. In one embodiment, the glasses50 can include a slot or hole 70 a so that the camera of the rear sideof the smartphone can see through the casing of the glasses 50. Inanother embodiment, the casing of the glasses 50 can include asee-through or clear panel to allow the smartphone 60 camera to seethrough the panel.

FIG. 1D illustrates an example of a smartphone 60, which can include acamera located on the rear side and also on the front side. The screenside 60 a of the smartphone will face forward toward the users eyes wheninserted into the glasses 50. As will be described below, the glasses50, in one embodiment, may include optics 80 that will reside betweenthe users eyes and the screen 60 a of the smartphone 60. The optics 80can be tailored to provide viewing of the content provided by thesmartphone screen 60 a and optimize the rendering, sizing, re-sizing,sharpness, prescription, and other distortion or non-distortionadjustments.

FIG. 2 illustrates an example of a user wearing the head mounted device(glasses) 50 which has received or is about to receive the smartphone60. The smartphone 60, in one embodiment, is connected to theelectronics 90 of the head mounted device 50 so that the optics 80 canbe synchronized with logic, software, firmware of the smartphone 60 forrendering of the images of the smartphone screen 60 a through the optics80 to the user. In one embodiment, the content being rendered by thesmartphone 60 will be virtual reality three-dimensional content, whichis delivered to the smartphone 60 in the form of an application that isdownloaded or in a streaming format from a website, service, serviceprovider, content provider, entertainment provider, or the like. In oneembodiment, the content rendered on the smartphone 60 is virtual realityrendering content which is shown in an example screen defined as the VRspace.

The front facing camera of the smartphone 60 may be monitoring thisfront view of the glasses 50 to assist the user in interacting with thescene, such as allowing the user 12 to place a hand 14 into the scene 16to interact with the content in the VR space. Thus, the user in the realworld space can place his hand into a view space of the front facingcamera, and this is detected by software operating on the smartphone toallow blending of the users hand 14 into the VR space. The users handcan be rendered into the VR space as the users actual hand or asaugmented reality, or as a blend of augmented and real world images.

FIG. 3A illustrates an example of the glasses 50, which are shown as ahousing that includes a clip or shelf 71 for receiving the smartphone60. It should be understood that the shape and ergonomics of the housingcan change depending on specific implementations. Thus, the exampleimplementations provided herein are just that, examples, which may bemodified so long as they function and provide the interactivity betweenthe devices, in accordance with the various embodiments.

In this particular example, the housing includes a clip 71 (or holder)that allows the front facing camera to freely view the front space orarea in the real world of the user. In one embodiment, as mentionedabove, the content may switch from a transparent mode to view outside ofthe virtual reality space, and back to the real world space. Thus, thefront facing camera 20 provides a real-world view when the user desiresto view the real world and exit the VR space. In another embodiment, thefront facing camera 20 can provide safety notifications to the userwhile the user is interacting in a VR space. For example, if the userwalks around a particular space and is dangerously approaching stairs, awall, or some other object, a notification can be provided to the userin the head mounted display as a message, notification, alarm, sound,tactile feedback, or the like.

The camera 20 is therefore useful for providing safety notifications ofthe real world space when the user becomes immersed in the virtualreality space. The user can also be provided with transitions out of thevirtual reality space into the real world space in partially transparentviews, fully transparent views, blends of fully and partiallytransparent views, or partial views that may show actual features thatare of interest to the user or may be dangerous when the user is wearingthe head mounted device that is enabled by a smartphone 60.

FIG. 3B illustrates an example of the head mounted device, whichincludes a headband, and the housing 70. The housing 70 is shown toinclude electronics 90 and optics 80, which sit between the location ofwhere the eyes of the user will reside and the front face of the screenof the smartphone 60, when placed in a slot, holder, clip, or connection71. A battery may also be provided with the housing to power theelectronics 90.

In another embodiment, a connector plug or wire 30 can be used toprovide power via a battery 32 disposed in the housing 70. In anotherembodiment, a battery of the smartphone 60 can operate electronics 90 ofthe head mounted device 50. As mentioned above, the smartphone 60 canalso be a tablet, a computer unit, a partial computer unit, or anyhybrid device with a display screen. In this example, the smartphone 60is connected to the receiver unit by way of a USB link connector 36. Thesmartphone 60 is therefore connected to the electronics 90 of thereceiver unit for handling information received from the smartphone 60,such as synchronization of image data with the operation of the optics80.

FIG. 4A illustrates an example of an electronics unit 90, which may beused in a head mounted device 50. In one example, the electronics unit90 can include a device interface 98 for connecting to a smartphone 60.The device interface 98 can be in communication with a microprocessor 94and memory 96. The microprocessor 94 can be coupled to the inertialsensors 91 that are part of the electronics unit 90 or receiver unit.The inertial sensors 91 may include accelerometers, gyroscopes,magnetometers, and combinations of one or more of the same orcombinations of those devices. In general, the inertial sensors 91provide data from movement and positioning of the housing 70 when wornby the user.

These motions and movements can be translated into information regardingposition that can be used for tracked and can be used to show or changethe images (i.e., image content) provided by the smartphone 60, inrespect to the virtual reality content being rendered or interfacedwith. For instance, as the user 12 moves his head around, differentimages and scenes can be provided by the display 60 a of the smartphone60 based on the content that is associated with those locations that theuser is looking at toward or away from. The microprocessor unit 94 canalso be in communication with a display shutter logic 92. Displayshutter logic 92 can be coupled to optics controller logic 93 that thencouples to the optics 80 of the receiver unit (i.e., housing). In oneembodiment, content can be provided to the smartphone 60 from a remoteserver in a formatted three-dimensional form or in a normal un-formattedform. For example, each frame can include a left and a right eye image,which is essentially the same image provided to both the left and theright eye and provided in a distorted format. In one embodiment, thisstream of frames can be downloaded in a streaming format from a serviceto the smartphone for display on the receiver unit via optics 80.

The display shutter logic 92 can configure the system to show the imagesto the respective eyes for received frames, in a sequential left, right,left, right, left, right, etc. In this manner, the left eye will see thecontent from the left part of the frame for a received frame, and thenthe right eye will see the content from the right part of the frame forthe next received frame, and then the left eye will see the left part ofa received frame, and then the right eye will see the content from theright part of the frame for the next frame, and so on. This isillustrated in more detail with reference to FIG. 4C.

FIG. 4B illustrates an example of a user 12 wearing a head mounteddevice with a smartphone 60 inserted therein. In this example, thesmartphone processing will receive a stream of frames from a cloudprocessing system which may be delivering three-dimensional games to theportable devices. In this illustration, each frame being delivered tothe smartphone will each include left and right eye image content. Asoftware module executed on the smartphone 60 or on electronics 90 ofthe head mounted device, or on both the electronics 90 of the headmounted device and the smartphone 60 will execute the receiving of theframes (e.g., a stream of frames from the cloud processing system). Thestream of frames can be delivered, for example at 30 frames per second.Other frame rates are possible, depending on the implementation.

As shown, the received image frame module will receive the images, andan up render (scale-up) module will scale up a selected left or rightpart of an image frame. The selected left or right image frame, beingscaled up, will be and provided to the display of the smartphone 60 fordisplay on the entire screen of the smartphone device 60, instead ofhaving the smartphone device show both the left and right eyes in asmaller less immersive format. Further, the software can include logicso that the head mounted device can select to shutter the left or righteye sequentially, so that the left or right eye, in continual sequence,will see the entire screen rendered by the smartphone. In one example,frames 0, 1, 2, 3, 4, and 5 are shown. These frames are being streamedfrom the cloud system and each frame will include a left and right eyeimage content. In accordance with one embodiment, for frame 0, thesoftware will select the left eye image content from the frame, and willrender up that left eye content for display on the portable device.

In synchronization, the software will provide instructions to a shuttersystem of the glasses which will open the shutter for the left eye sothat the user views the entire screen for the left eye at the time whenthe scaled up image is provided to the entire display screen of thephone. The same process will happen with frame 1, except that the righteye content is scaled up, shown on the display screen, and the shutterwill show the content to the right eye of the user. This process willcontinue sequentially so as to provide a three-dimensional effect to thecontent being displayed by the smartphone. In one embodiment, thesoftware the smartphone is essentially providing for a frame select, andup render, and shutter select instructions to the glasses.

In an alternate embodiment, the image data received from the source willprovide full images, which avoids the need to up-render. In this case,the shutter logic will simply show the content to the left and righteyes, switching successively, to provide a three-dimensional effect. Assuch, the 3D effect can be processed with or without pre-formatting ofthe image content that is delivered or displayed by the smartphone 60via the optics 80. In this embodiment, therefore, the shutter logicsimply synchronizes with the frame rate of images rendered by the screenof the smartphone, and the left/right shuttering provides for thenecessary three-dimensional effect. In some embodiments, the image data(e.g., interactive video) is sent for use with the shutter system and inothers, the image data is simply sent and the shuttering is done by theelectronics 90 of the housing 70.

FIG. 4C illustrates an example of the stream arriving at 30 frames persecond arriving from a streaming service. In this example, with theshutter system of the glasses, each eye sees 15 frames per second. Attime 0, the phone will display the left eye image content that has beenscaled up and the shutter will open the left eye for viewing of thecontent. At time 1, the software will select the right eye content fromthe frame scaled up and the shutter will allow the right eye to view theentire screen of the phone display. This process will continue for time2, time 3 and so on. In other embodiments, the frame rate can beincreased for the stream of images coming into the smartphone 60,depending on bandwidth. In one embodiment, if the frame rate of thestream frames is 60 frames per second, each eye will see 30 frames persecond. In essence, the content can be made to be rich three-dimensionalcontent by simply using a shutter system that will select parts of animage of the frame and then display a scaled up version to the displayscreen. As noted above, scale-up is optional, and the system can operateon full received image frames and shuttering and display via the optics80. The frame rate, in some embodiments can also be 120, 240, or moreframes per second, depending on the bandwidth and processing power.

In still another example, with reference to FIGS. 4B and 4C, athree-dimensional video stream is transmitted to a head mounted display(i.e., the screen 60 a of a smartphone 60). Each frame of thethree-dimensional video stream includes both a left eye image and aright eye image. A given frame is processed to extract either the lefteye image or the right eye image, but not both. For a given frame, theextracted image (left eye or right eye) is up-rendered to fit thedisplay size of the head mounted display, and the shutter of the headmounted display is set to allow display of the portion of the givenframe (either left eye or right eye) that is currently extracted andup-rendered. Then, the extracted and up-rendered portion of the givenframe is displayed.

For sequential frames, the extraction, up-rendering, shutter setting,and displaying operations alternate between the left eye image and theright eye image, such that for a given frame only the left eye imageportion or the right eye image portion is processed, with the next framehaving the other of the left eye image portion or right eye imageportion processed, and so on. With a normal frame rate, the processingof only the left eye image portion or right eye image portion of a givenframe in an alternating manner is not discernable to the human opticsystem, and provides for improved efficiency in processing thethree-dimensional video stream. Again, the video stream can also providenon-3D image data, which is the given 3D effect by the shuttering. Also,in some embodiments, the image data need not be up-rendered, and thefull size images can be shuttered.

FIG. 5 illustrates an example of an electronics unit 90, which mayinclude other functionality, circuitry, firmware, logic, processors,memory, etc. These features are only exemplary and some systems mayinclude more or fewer devices depending on the implementation. In theone example provided in FIG. 5, the electronics unit 90 may be part ofthe glasses 50. The electronics unit 90 will include a processor 94which can include a graphics processor or a GPU, or general processor.In some embodiments, multiple processors may be provided, or multicoreprocessors. The processors 94 can be in communication with inertialsensors 91 that may be part of the glasses or head mounted device. Theinertial sensors 91 can collect information for motion, movement, deadreckoning, and tracking. This information can be shared with theprocessor 94.

In another embodiment, a plurality of photo diodes are disposed on asurface of the housing. The photo diodes are used for tracking positionof the housing using at least one light emitting device positioned inspace/room where the housing is to be tracked. For example, the photodiodes can be disposed on the surface of the housing, e.g., with knownphysical separations to one another. The light emitter can, for example,shoot out light in various directions so that some of the light willcontact/reach the photo diodes. A timing profile is calculated orcalibrated, so that timing is known or synced with the detection andemitting. In this manner, it is possible to identify when certain of thephoto diodes detect light, and because the location of the emitter(e.g., one or two rotating emitters paced in a room) and timing isknown, it is possible to identify the position of the housing having thephoto diodes. As the emitting and detecting is on-going during the useof the HMD system, tracking of the HMD/housing is facilitated.

The processor can also control the shutter control logic 92, which thencan interface with controllable optics 93, which can control the leftand right eye optics 80 of the glasses. In one embodiment, the processorcan also be coupled to a frame buffer 95 and memory 96, or simply memory96. The frame buffer 95 can also be in communication with an imagerendering logic 97 that also provides image data to the left and righteye of the optics 80. For example, the head mounted device (glasses) 50can also receive a separate feed of content or data from the Internetwhich can be blended with the content received by the smartphone 60 anddisplayed by the smartphones screen 60 a. In one embodiment, the contentdisplayed by the smartphone 60 can be augmented with additional contentprovided by the head mounted display 50. In one embodiment, the contentobtained by the head mounted device (glasses) 50, can be blended,multiplex, or geared with content received by and displayed by thesmartphone. For example, content can be switched between that receivedby the head mounted device and that received by the smartphone 60.

In other embodiments, depending on the content being displayed,overlays, communication information, second channel chat feeds, socialnetwork data, picture in a picture display data or other content can beprovided to the left or right or both eyes through the optics of theglasses and can be blended or replaced or interlaced with data providedby the smartphone device 60. In one embodiment, the smartphone device 60and the electronics unit 90 can include a network interface 99 and adevice interface 98, which will allow communication with the Internetand provide access to various content sites. The content site canprovide content to the devices for VR rendering, image production,communication, social network sharing, virtual reality tours,destination touring content, cloud gaming content, cloud entertainmentcontent, social media content, streaming servers, nearby head mounteddisplay players, and other information.

In one embodiment, an active shutter 3D system (e.g., alternate framesequencing, alternate image, AI, alternating field, field sequential oreclipse method) may be used to display stereoscopic 3D images. In oneconfiguration, a system works by only presenting the image intended forthe left eye while blocking the right eye's view, then presenting theright-eye image while blocking the left eye, and repeating this sorapidly that the interruptions do not interfere with the perceivedfusion of the two images into a single 3D image.

Some active shutter 3D systems use liquid crystal shutter glasses (alsocalled “LC shutter glasses” or “active shutter glasses”). OLED screensare also usable. Each eye's glass contains a liquid crystal layer whichhas the property of becoming opaque when voltage is applied, beingotherwise transparent. The glasses are controlled by a timing signalthat allows the glasses to alternately block one eye, and then theother, in synchronization with the refresh rate of the screen. Thetiming synchronization to the video equipment may be achieved via awired signal, or wirelessly by either an infrared or radio frequency(e.g., Bluetooth, DLP link) transmitter. Active shutter 3D systems canalso be implemented to present 3D films in some theaters, and they canbe used to present 3D images on CRT, plasma, LCD, 3D images on HMDsystems, VR display systems, smartphones, game consoles, screens,displays, and combinations thereof.

FIG. 6 illustrates an example of a multiplayer or multiuser environment,where multiple users may be using a head mounted device 50 with asmartphone 60. In one embodiment, the smartphones 60 can communicatewith a cloud system, which can provide streaming VR content. In oneembodiment, the users may become close in proximity to each other whichmay allow the systems and software associated with the smartphones andVR content mixing to switch modes of communication. For example, if apair of users are sharing a virtual reality site or location orinteraction zone, some or all of the content can be communicated betweenthe users via a peer-to-peer network (i.e., in a paired mode), which isa mode change from peer to cloud network.

The switching can occur dynamically over time when the proximity of theuser's is detected, or per user setting or request. Latency can also bereduced by enabling local peer-to-peer communication. Switch topeer-to-peer can be for select data, e.g., a data channel, while theinteractive video is being streamed to both users. For instance, localcommunication such as messaging, chat, audio channels, and notificationscan be transferred via peer-to-peer, while the interactive data isstreamed from the server to each user. In some cases, some streamingdata can be sent one user and this data is shared to the other use,e.g., when one user has a higher bandwidth connection than the other.Accordingly, these embodiments would allow users to share VR spaces andcommunicate data between each other in a mixed mode fashion, which canallow both communication to the cloud system and peer-to-peercommunication, near field communication, Wi-Fi communication, Bluetoothcommunication, or any combinations of other connected wired or wirelesslinks.

FIG. 7 illustrates another example of a head mounted device 50 which caninclude the slot 71 for receiving a smartphone 60. As discussed above,the smartphone 60 can be connected to the head mounted device or housing70 by way of a connector 40. The connector 40 can be a wired connectionor a wireless connection. Once the smartphone 60 is inserted into aholder 71 of the housing, the smartphone 60 can allow its front facingcamera to face forward. In one embodiment, the housing is designed sothat the camera of the smartphone can view forward without beingblocked. The head mounted device 50 can also include electronics 90 andoptics 80 as described above. In this embodiment, the head mounteddevice can also include two additional cameras, such as a left camera ora right camera on the housing of the HMD 50. These additional camerascan provide for three-dimensional viewing of content in front of theuser when the user decides to flip between virtual reality scenes toreal life and real world scenes in front of the user, e.g., in a passthrough mode (or transport or semi-transport mode).

In this example, the electronics 90 of the head mounted device 50 caninclude a wireless network interface 99 or wired interface they cancommunicate with the Internet. The smartphone 60, shown inserted intothe holder 71 of the housing and can also include its own Internetconnection. The camera of the smartphone is front forward facing, butcan include other cameras as well. Of course, the smartphone 60 caninclude its own inertial sensors (not shown). In addition, the headmounted device 50 can include inertial sensors 91 that are also incommunication with a processor 94 of the head mounted device 50. Theinertial sensors 91 of the head mounted device 50 can be incommunication with the inertial sensors of the smartphone.

In one embodiment, blending, mixing, or dead reckoning can occur betweendata collected from the inertial sensors 91 of the head mounted device50 and the smartphone 60, to avoid false positives, jerky movements,unexpected motions, or triggers that would be unnatural when viewingcontent in a VR scene. In this example, the processor 94 is incommunication with shutter control 92 and camera select right-left whichcommunicate with optics control 93 for the right and left eyes. Memoryand/or a frame buffer 95 can also be provided so that the processor 94can provide different data to the display screen than that provided bythe display screen of the phone or a blended, multiplex, overlay,message overlay, social connection overlay, and the like can be providedby the head mounted device in addition to the content provided by thissmartphone. In one embodiment, the processor can act to select thecamera select right/left, so that the front facing left and rightcameras of the head mounted device can be activated when the user wishesto see through the device and look at the real world.

Looking at the real world through two cameras can occur by way of ashutter control 92 that flips between left and right eye sequentiallyand continuously. This functionality will provide for the imagesassociated with the real world to be more lifelike and includethree-dimensional depth perspectives not provided by single camera frontfacing systems. In still another embodiment, the front facing cameras ofthe head mounted device 50 can assist for safety features, such as toadvise users when the users are coming close to walls, objects, stairs,or unsafe zones.

In still another example, with reference to FIGS. 4B and 4C, athree-dimensional video stream is transmitted to a head mounted display.Each frame of the three-dimensional video stream includes both a lefteye image and a right eye image. A given frame is processed to extracteither the left eye image or the right eye image, but not both. For agiven frame, the extracted image (left eye or right eye) is up-renderedto fit the display size of the head mounted display, and the shutter ofthe head mounted display is set to allow display of the portion of thegiven frame (either left eye or right eye) that is currently extractedand up-rendered. Then, the extracted and up-rendered portion of thegiven frame is displayed.

For sequential frames, the extraction, up-rendering, shutter setting,and displaying operations alternate between the left eye image and theright eye image, such that for a given frame only the left eye imageportion or the right eye image portion is processed, with the next framehaving the other of the left eye image portion or right eye imageportion processed, and so on. With a normal frame rate, the processingof only the left eye image portion or right eye image portion of a givenframe in an alternating manner is not discernable to the human opticsystem, and provides for improved efficiency in processing thethree-dimensional video stream.

In other embodiments, cameras may be provided separately from the HMD orthe smartphone. The cameras can be directed toward the HMD to track theHMD. In some embodiments, combinations of cameras, which may includecameras on the HMD, cameras on the phone, and cameras viewing the HMD(e.g., as connected to a computer) may be used together. As such, itshould be understood that the embodiments or parts of the embodimentsdescribed with reference to FIGS. 1A-7 may be combined with parts,components, features or embodiments described with reference to FIGS.8-12, to define various hybrid embodiments.

With the foregoing in mind, additional embodiments are described withreference to FIGS. 8-12.

In one example, the HMD worn by a user provides the user access to viewrich multimedia content, which can include video games, movies, internetcontent, and other types of interactive and non-interactive content. Thetracking of the HMD is performed using a combination of systems. Thesystems include, without limitation, inertial sensors in the HMD andoptical tracking using one or more cameras. A camera used in opticaltracking can capture video of the user wearing the HMD, so when the usermoves around with the HMD the video frames can be analyzed to determineposition, orientation and movements of the HMD. Broadly speaking, somecontent presented by the HMD is dynamically dependent on movement of theHMD.

For example, if the HMD provides a view into a scene, the user is ableto naturally move his or her head to view other parts of the scene. In avideo gaming example, a user wearing the HMD can move his or her head inany direction to move about in and around a virtual scene. In oneembodiment, the virtual scene is rendered in a rich three dimensional(3D) format. Consequently, in order to smoothly render content in theHMD, the movement of the HMD will be tracked with high fidelity.

In one configuration of an HMD, shown in FIGS. 8A and 8B, is configuredto communicate with a client system 106, which renders the contentpresented to the HMD. The content (e.g., game, movie, video, audio,images, multimedia, etc.), in some embodiments may be streamed from aremote server or servers using cloud gaming infrastructure. In someexamples, the content is downloaded to the client system 106 forrendering and then transferred to the HMD.

As noted above, the tracking may include the use of inertial sensorsthat are disposed within the HMD. Example inertial sensors include oneor more accelerometers and one or more gyroscopes. Some implementationsmay include more or less inertial sensors. In addition to inertialsensors, the HMD can be tracked using a camera. The HMD is, in oneembodiment, configured with several lights (e.g., light emitting diodes(LEDs)), which act as markers. The markers can then be easily identifiedby analyzing, by the client system, one or more video frames captured bythe camera. In one configuration, the HMD includes four LEDs on the fourcorners of the front unit (e.g., also referred to herein as the opticsblock) and two LEDs on the rear section.

The front unit 102 a, in one example, includes a front face and a sideface on each side, wherein the front face and side faces define asubstantially continuous surface. In various examples provided herein,the front LEDs are defined in a housing of the front unit 102 a, anddisposed with transparent plastic that can illuminate when the LEDs areturned on. Further, in some embodiments, the front LEDs are configuredto be partially disposed on the front surface and partially on the sidesurface, to define a partial L-shape, or curved L-shape, or boomerangshape, or a curved rectangle, or curved line, or a spot, or circle, or apattern, or combinations thereof.

This shape allows for tracking of the front unit 102 a when the user isdirectly facing the camera 108 and when the user starts to turn awayfrom direct facing of the camera 108. As the user faces to the side andfurther away from the directly facing the camera, the front LEDs will bevisible until only the LEDs on one side of the front are visible and oneof the LEDs on the backside are visible. This is the transition from thefront LEDs to the front and back LEDs. Due to this transition, as notedabove, the separation distance between the front and back LEDs isneeded, so that accurate tracking can proceed.

Still further, when the user wearing the HMD is facing the camera, thecamera should be able to view all four LEDs. The separation of the fourfront LEDs is known to the client system 106. For example, a geometricmodel of the HMD can be accessed by programs executed on the clientsystem 106, to determine depth (relative to the camera) and orientationof the user's head when wearing the HMD. For instance, because the fourLEDs are, in one embodiment, disposed on the corners (e.g., outlining arectangular shape), it is possible to determine from the captured videoframes if the user is viewing down, up or to the sides.

However, because the interactive content that can be rendered in the HMDcan be virtually boundless, a user is able to view and interact with thevirtual scene in most every dimension. A user wearing an HMD, therefore,may decide to turn his or her head in any direction, which is notnecessarily always forward facing with respect to the camera. In fact,depending on the content rendered (e.g., immersive interactive videogames, moves, virtual tours, clips, audio, and combinations thereof),users will many times be facing to the sides of the camera and directlyaway from the camera.

During such interactive sessions, the camera tracking the HMD will gofrom seeing the front four LEDs to sometimes seeing the side of two ofthe front LEDs and also one of the rear LEDs. Although the front fourLEDs remain in a fixed relative orientation, based on the geometricmodel of the HMD, the rear LEDs may change in position depending on anadjusted setting of a headband of the HMD. For instance, if a user witha smaller head adjusts the headband to fit, the distance between thefront LEDs and the rear LED (e.g., when viewed from the side) will becloser, relative to an adjusted setting for a larger head of anotheruser.

To account for the changes in headband adjustments, a process isconfigured to calibrate the geometric model of the HMD, so that theseparation between the front LEDs and the rear LED (e.g., when viewedfrom the side when the user turns his or her head away from normal tothe camera) can be used to accurately render scene content to the HMDand provide the scenes from the desired perspective, angle and/ororientation. In one implementation, the geometric model is a computermodel, which stores/contains dimensions and/or three-dimensionaloutlines of the HMD 102, similar to what a computer aided design (CAD)drawing may show. However, the geometric model is not displayed as adrawing, but instead is stored as a data set, that is accessible bygames, or movies, or software, or firmware, or hardware, or combinationsthereof to enable accurate tracking.

Still by way of example, the three-dimensional outlines of the HMD 102can include, in one embodiment, outlines of each shape of the HMD andthe shapes of the LED regions, the locations of the LEDs relative toshapes in the outline, the angles and contours of the physicalstructures of the HMD, and data sets that define measurements of thefeatures and constructs of the HMD. In particular, the geometric modelmay include dimensional data that define the exact relative placement ofthe LEDs on the front unit 102 a. However, because the rear LEDs arecoupled to a headband that is adjustable, the separate distance must beupdated during calibration, so that the geometric model can be updatedwith a more accurate distance between the front and rear LEDs.

In one embodiment, the calibration process is configured to initiateafter analysis of the video frames determines that one of the rear LEDsis visible (e.g., starting from when only the front LEDs are visible).For example, at the start of a session (e.g., game play or interactivesession), it is common that the user will face the HMD toward thecamera. At some point, the user will turn his head away from the camera,which will expose at least one of the rear LEDs. At this point, theanalysis of the video frames, which is ongoing, will detect theappearance of the rear LED.

In one embodiment, the process will analyze several frames as the usercontinues to move to associate the visible rear LED and the visiblefront LEDs with inertial data. The inertial data present for each frame,for example, is used to associate an estimated separation distancebetween the visible rear LED and the visible front LEDs. In one example,gyroscope data from the HMD is used to determine the rotation motion bythe user's head, as the HMD moves. Further, by way of example,accelerometer data from the HMD is used to determine movement, such asposition (e.g., tilt/pitch) and rotation.

Thus, using the image data from the captured video frames of the HMD(i.e., when the rear and front LEDs are visible), the inertial data incombination with the image data will render an estimated separationdistance between the front LEDs and the rear LED, for the current sizesetting of the headband of the HMD. This data is then used to calibratethe geometric model of the HMD, which includes the estimated separationdistance. In one embodiment, the calibration can be updated from time totime, and can also be calibrated independently for each side of the HMD.

Once the calibration to the geometric model of the HMD is complete, theuser may proceed to interact during the session. However, once thesession is done, it is possible that a different user may wish access tothe HMD. At such time, it is also likely that the new user will adjustthe headband of the HMD to another size, which will cause a change inthe actual separation distance between the front LEDs and the rear LEDs.In one embodiment, a new session can begin, using either the priorupdated calibration or the dimensions from the original geometric model.

At the start, the game play or interactive session will proceedsmoothly, wherein the scene presented in the HMD will render based onmovements in the user's head. This will be so, while the user wearingthe HMD is facing forward toward the camera, wherein the fixedseparation between the four LEDs in the front of the HMD are known.However, once the user turns away from the camera and the rear LED isfound, the system, without automatic calibration, would see a jump orpop in the content rendered in the HMD. This is so, as the tracking ofthe HMD, which utilizes marker tracking of the LEDs to identifyposition, would be out of sync with the true position of the HMD.

In one embodiment, a determination as to whether re-calibration isneeded is performed each time the user faces the HMD to the side,wherein the front and rear LEDs become visible (i.e., coming from whenonly the front or only the rear LEDs are visible). In one example, ifthe calibration occurred for a current session, and the session ison-going with a current calibration, the system will run a calibrationin the background to determine if the current calibration is stillwithin a pre-defined tolerance margin. For example, if the same useradjusted the headband during game play, or took off the HMD for a minorsize adjustment, or some other person tried on the HMD momentarily, thenthe actual separation between the front and the rear would be differentthan what was used to estimate the separation during an initialcalibration.

The tolerance margin is configured or chosen so that if the newbackground calibration shows that rendering glitches, skips or popswould likely occur (e.g., in the video images rendered in the HMD), thenthe new calibration should become the current calibration.

In still another embodiment, the HMD will include a headband adjustmentdetector that will set a flag. The flag can be read by the system and/orgame executing, which can be used to require recalibration of thegeometric model. For instance, if the user adjusts the headband duringuse of the HMD, the system can be alerted via the flag that thecalibration should be re-run. The same may be true if the adjustmentoccurred because another user tried on the HMD, even if the same sessionis in progress. In still further embodiments, a flag can be generatedupon the start of a new session or when the system detects that the HMDhas been still or has not moved for some time. Such indicators can beviewed as a possibility that the headband may have been adjusted, beforethe new session occurred or even during a session.

FIG. 8A illustrates a system for interactive gameplay of a video game,in accordance with an embodiment of the invention. A user 100 is shownwearing a head-mounted display (HMD) 102. The HMD 102 is worn in amanner similar to glasses, goggles, or a helmet, and is configured todisplay a video game or other content to the user 100. The HMD 102 isconfigured to provide an immersive experience to the user by virtue ofits provision of display mechanisms (e.g., optics and display screens)in close proximity to the user's eyes and the format of the contentdelivered to the HMD. In one example, the HMD 102 can provide displayregions to each of the user's eyes which occupy large portions or eventhe entirety of the field of view of the user.

In one embodiment, the HMD 102 can be connected to a computer 106. Theconnection to computer 106 can be wired or wireless. The computer 106can be any general or special purpose computer, including but notlimited to, a gaming console, personal computer, laptop, tabletcomputer, mobile device, cellular phone, tablet, thin client, set-topbox, media streaming device, etc. In some embodiments, the HMD 102 canconnect directly to the internet, which may allow for cloud gamingwithout the need for a separate local computer. In one embodiment, thecomputer 106 can be configured to execute a video game (and otherdigital content), and output the video and audio from the video game forrendering by the HMD 102. The computer 106 is also referred to herein asa client system 106 a, which in one example is a video game console.

The computer may, in some embodiments, be a local or remote computer,and the computer may run emulation software. In a cloud gamingembodiment, the computer is remote and may be represented by a pluralityof computing services that may be virtualized in data centers, whereingame systems/logic can be virtualized and distributed to user over anetwork.

The user 100 may operate a controller 104 to provide input for the videogame. In one example, a camera 108 can be configured to capture image ofthe interactive environment in which the user 100 is located. Thesecaptured images can be analyzed to determine the location and movementsof the user 100, the HMD 102, and the controller 104. In one embodiment,the controller 104 includes a light (or lights) which can be tracked todetermine its location and orientation. Additionally, as described infurther detail below, the HMD 102 may include one or more lights whichcan be tracked as markers to determine the location and orientation ofthe HMD 102 in substantial real-time during game play.

The camera 108 can include one or more microphones to capture sound fromthe interactive environment. Sound captured by a microphone array may beprocessed to identify the location of a sound source. Sound from anidentified location can be selectively utilized or processed to theexclusion of other sounds not from the identified location. Furthermore,the camera 108 can be defined to include multiple image capture devices(e.g. stereoscopic pair of cameras), an IR camera, a depth camera, andcombinations thereof.

In some embodiments, computer 106 can execute games locally on theprocessing hardware of the computer 106. The games or content can beobtained in any form, such as physical media form (e.g., digital discs,tapes, cards, thumb drives, solid state chips or cards, etc.) or by wayof download from the Internet, via network 110. In another embodiment,the computer 106 functions as a client in communication over a networkwith a cloud gaming provider 112. The cloud gaming provider 112 maymaintain and execute the video game being played by the user 100. Thecomputer 106 transmits inputs from the HMD 102, the controller 104 andthe camera 108, to the cloud gaming provider, which processes the inputsto affect the game state of the executing video game. The output fromthe executing video game, such as video data, audio data, and hapticfeedback data, is transmitted to the computer 106. The computer 106 mayfurther process the data before transmission or may directly transmitthe data to the relevant devices. For example, video and audio streamsare provided to the HMD 102, whereas a vibration feedback command isprovided to the controller 104.

In one embodiment, the HMD 102, controller 104, and camera 108, maythemselves be networked devices that connect to the network 110 tocommunicate with the cloud gaming provider 112. For example, thecomputer 106 may be a local network device, such as a router, that doesnot otherwise perform video game processing, but facilitates passagenetwork traffic. The connections to the network by the HMD 102,controller 104, and camera 108 may be wired or wireless. In someembodiments, content executed on the HMD 102 or displayable on a display107, can be obtained from any content source 120. Example contentsources can include, for instance, internet websites that providedownloadable content and/or streaming content. In some examples, thecontent can include any type of multimedia content, such as movies,games, static/dynamic content, pictures, social media content, socialmedia websites, etc.

As will be described below in more detail, a player 100 may be playing agame on the HMD 102, where such content is immersive 3D interactivecontent. The content on the HMD 102, while the player is playing, can beshared to a display 107. In one embodiment, the content shared to thedisplay 107 can allow other users proximate to the player 100 or remoteto watch along with the user's play. In still further embodiments,another player viewing the game play of player 100 on the display 107may participate interactively with player 100. For example, a userviewing the game play on the display 107 may control characters in thegame scene, provide feedback, provide social interaction, and/or providecomments (via text, via voice, via actions, via gestures, etc.,) whichenables users that are not wearing the HMD 102 to socially interact withplayer 100, the game play, or content being rendered in the HMD 102.

FIG. 8B illustrates a head-mounted display (HMD), in accordance with anembodiment of the invention. As shown, the HMD 102 includes a pluralityof lights 200A-H, J and K (e.g., where 200K and 200J are located towardthe rear or backside of the HMD headband). Each of these lights may beconfigured to have specific shapes and/or positions, and can beconfigured to have the same or different colors. The lights 200A, 200B,200C, and 200D are arranged on the front surface of the HMD 102. Thelights 200E and 200F are arranged on a side surface of the HMD 102. Andthe lights 200G and 200H are arranged at corners of the HMD 102, so asto span the front surface and a side surface of the HMD 102. It will beappreciated that the lights can be identified in captured images of aninteractive environment in which a user uses the HMD 102.

Based on identification and tracking of the lights, the location andorientation of the HMD 102 in the interactive environment can bedetermined. It will further be appreciated that some of the lights mayor may not be visible depending upon the particular orientation of theHMD 102 relative to an image capture device. Also, different portions oflights (e.g. lights 200G and 200H) may be exposed for image capturedepending upon the orientation of the HMD 102 relative to the imagecapture device. In some embodiments, inertial sensors are disposed inthe HMD 102, which provide feedback regarding positioning, without theneed for lights. In some embodiments, the lights and inertial sensorswork together, to enable mixing and selection of position/motion data.

In one embodiment, the lights can be configured to indicate a currentstatus of the HMD to others in the vicinity. For example, some or all ofthe lights may be configured to have a certain color arrangement,intensity arrangement, be configured to blink, have a certain on/offconfiguration, or other arrangement indicating a current status of theHMD 102. By way of example, the lights can be configured to displaydifferent configurations during active gameplay of a video game(generally gameplay occurring during an active timeline or within ascene of the game) versus other non-active gameplay aspects of a videogame, such as navigating menu interfaces or configuring game settings(during which the game timeline or scene may be inactive or paused). Thelights might also be configured to indicate relative intensity levels ofgameplay. For example, the intensity of lights, or a rate of blinking,may increase when the intensity of gameplay increases.

The HMD 102 may additionally include one or more microphones. In theillustrated embodiment, the HMD 102 includes microphones 204A and 204Bdefined on the front surface of the HMD 102, and microphone 204C definedon a side surface of the HMD 102. By utilizing an array of microphones,sound from each of the microphones can be processed to determine thelocation of the sound's source. This information can be utilized invarious ways, including exclusion of unwanted sound sources, associationof a sound source with a visual identification, etc.

The HMD 102 may also include one or more image capture devices. In theillustrated embodiment, the HMD 102 is shown to include image captureddevices 202A and 202B. By utilizing a stereoscopic pair of image capturedevices, three-dimensional (3D) images and video of the environment canbe captured from the perspective of the HMD 102. Such video can bepresented to the user to provide the user with a “video see-through”ability while wearing the HMD 102. That is, though the user cannot seethrough the HMD 102 in a strict sense, the video captured by the imagecapture devices 202A and 202B can nonetheless provide a functionalequivalent of being able to see the environment external to the HMD 102as if looking through the HMD 102.

Such video can be augmented with virtual elements to provide anaugmented reality experience, or may be combined or blended with virtualelements in other ways. Though in the illustrated embodiment, twocameras are shown on the front surface of the HMD 102, it will beappreciated that there may be any number of externally facing cameras ora single camera can be installed on the HMD 102, and oriented in anydirection. For example, in another embodiment, there may be camerasmounted on the sides of the HMD 102 to provide additional panoramicimage capture of the environment.

FIG. 9 illustrates one example of gameplay using the client system 106that is capable of rendering the video game content to the HMD 102 ofuser 100. In this illustration, the game content provided to the HMD isin a rich interactive 3-D space. As discussed above, the game contentcan be downloaded to the client system 106 or can be executed in oneembodiment by a cloud processing system. Cloud gaming service 112 caninclude a database of users 140, which are allowed to access particulargames, share experiences with other friends, post comments, and managetheir account information.

The cloud gaming service can also store game data 150 for specificusers, which may be usable during gameplay, future gameplay, sharing toa social media network, or for storing trophies, awards, status,ranking, etc. Social data 160 can also be managed by cloud gamingservice 112. The social data can be managed by a separate social medianetwork, which can be interfaced with cloud gaming service 112 over theInternet 110. Over the Internet 110, any number of client systems 106can be connected for access to the content and interaction with otherusers.

Continuing with the example of FIG. 9, the three-dimensional interactivescene viewed in the HMD can include gameplay, such as the charactersillustrated in the 3-D view. One character, e.g. P1 can be controlled bythe user 100 that is wearing the HMD 102. This example shows abasketball scene between two players, wherein the HMD user 100 isdunking a ball on another character in the 3-D view. The other charactercan be an AI (artificial intelligence) character of the game, or can becontrolled by another player or players (Pn). User 100, who is wearingthe HMD 102 is shown moving about in a space of use, wherein the HMD maymove around based on the user's head movements and body positions. Thecamera 108 is shown positioned over a display screen in the room,however, for HMD use, the camera 108 can be placed in any location thatcan capture images of the HMD 102. As such, the user 102 is shown turnedat about 90 degrees from the camera 108 and the display 107, as contentrendered in the HMD 102 can be dependent on the direction that the HMD102 is positioned, from the perspective of the camera 108. Of course,during HMD use, the user 100 will be moving about, turning his head,looking in various directions, as may be needed to take advantage of thedynamic virtual scenes rendered by the HMD.

FIG. 10 illustrates a user wearing the HMD 102, during use, inaccordance with one embodiment. In this example, it is shown that theHMD is tracked 802 using image data obtained from captured video framesby the camera 108. Additionally, it is shown that the controller canalso be tracked 804 using image data obtained from captured video framesby the camera 108. Also shown is the configuration where the HMD isconnected to the computing system 106 via a cable 806. In oneembodiment, the HMD obtains power from the same cable or can connect toanother cable. In still another embodiment, the HMD can have a batterythat is rechargeable, so as to avoid extra power cords.

With reference to FIG. 11, a diagram is shown illustrating examplecomponents of a head-mounted display 102, in accordance with anembodiment of the invention. It should be understood that more or lesscomponents can be included or excluded from the HMD 102, depending onthe configuration and functions enabled. The head-mounted display 102may include a processor 900 for executing program instructions. A memory902 is provided for storage purposes, and may include both volatile andnon-volatile memory. A display 904 is included which provides a visualinterface that a user may view.

The display 904 can be defined by one single display, or in the form ofa separate display screen for each eye. When two display screens areprovided, it is possible to provide left-eye and right-eye video contentseparately. Separate presentation of video content to each eye, forexample, can provide for better immersive control of three-dimensional(3D) content. As described above, in one embodiment, the second screen107 is provided with second screen content of the HMD 102 by using theoutput for one eye, and then formatting the content for display in a 2Dformat. The one eye, in one embodiment, can be the left-eye video feed,but in other embodiments it can be the right-eye video feed.

A battery 906 may be provided as a power source for the head-mounteddisplay 102. In other embodiments, the power source can include anoutlet connection to power. In other embodiments, an outlet connectionto power and a battery 906 may be provided. A motion detection module908 may include any of various kinds of motion sensitive hardware, suchas a magnetometer 910, an accelerometer 912, and a gyroscope 914.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis (e.g., six-axis)models are able to detect magnitude and direction of the acceleration indifferent directions. The accelerometer is used to sense inclination,vibration, and shock. In one embodiment, three accelerometers 912 areused to provide the direction of gravity, which gives an absolutereference for two angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of the head-mounted display. In one embodiment, threemagnetometers 910 are used within the head-mounted display, ensuring anabsolute reference for the world-space yaw angle. In one embodiment, themagnetometer is designed to span the earth magnetic field, which is ±80microtesla. Magnetometers are affected by metal, and provide a yawmeasurement that is monotonic with actual yaw. The magnetic field may bewarped due to metal in the environment, which causes a warp in the yawmeasurement. If necessary, this warp can be calibrated using informationfrom other sensors such as the gyroscope or the camera. In oneembodiment, accelerometer 912 is used together with magnetometer 910 toobtain the inclination and azimuth of the head-mounted display 102.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes 914 provide information about movement across the respectiveaxis (x, y and z) based on inertial sensing. The gyroscopes help indetecting fast rotations. However, the gyroscopes can drift overtimewithout the existence of an absolute reference. This requires resettingthe gyroscopes periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of an object, accelerometer, magnetometer, etc.

A camera 916 is provided for capturing images and image streams of areal environment. More than one camera (optionally) may be included inthe head-mounted display 102, including a camera that is rear-facing(directed away from a user when the user is viewing the display of thehead-mounted display 102), and a camera that is front-facing (directedtowards the user when the user is viewing the display of thehead-mounted display 102). Additionally, a depth camera 918 may beincluded in the head-mounted display 102 for sensing depth informationof objects in a real environment.

The head-mounted display 102 includes speakers 920 for providing audiooutput. Also, a microphone 922 may be included for capturing audio fromthe real environment, including sounds from the ambient environment,speech made by the user, etc. The head-mounted display 102 includestactile feedback module 924 for providing tactile feedback to the user.In one embodiment, the tactile feedback module 924 is capable of causingmovement and/or vibration of the head-mounted display 102 so as toprovide tactile feedback to the user.

LEDs 926 are provided as visual indicators of statuses of thehead-mounted display 102. For example, an LED may indicate batterylevel, power on, etc. A card reader 928 is provided to enable thehead-mounted display 102 to read and write information to and from amemory card. A USB interface 930 is included as one example of aninterface for enabling connection of peripheral devices, or connectionto other devices, such as other portable devices, computers, etc. Invarious embodiments of the head-mounted display 102, any of variouskinds of interfaces may be included to enable greater connectivity ofthe head-mounted display 102.

A WiFi module 932 may be included for enabling connection to theInternet via wireless networking technologies. Also, the head-mounteddisplay 102 may include a Bluetooth module 934 for enabling wirelessconnection to other devices. A communications link 936 may also beincluded for connection to other devices. In one embodiment, thecommunications link 936 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 936 mayutilize any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 938 are included to provide an input interface forthe user. Any of various kinds of input interfaces may be included, suchas buttons, gestures, touchpad, joystick, trackball, etc. An ultra-soniccommunication module 940 may be included in head-mounted display 102 forfacilitating communication with other devices via ultra-sonictechnologies.

Bio-sensors 942 are included to enable detection of physiological datafrom a user. In one embodiment, the bio-sensors 942 include one or moredry electrodes for detecting bio-electric signals of the user throughthe user's skin, voice detection, eye retina detection to identifyusers/profiles, etc.

The foregoing components of head-mounted display 102 have been describedas merely exemplary components that may be included in head-mounteddisplay 102. In various embodiments of the invention, the head-mounteddisplay 102 may or may not include some of the various aforementionedcomponents. Embodiments of the head-mounted display 102 may additionallyinclude other components not presently described, but known in the art,for purposes of facilitating aspects of the present invention as hereindescribed.

It will be appreciated by those skilled in the art that in variousembodiments of the invention, the aforementioned handheld device may beutilized in conjunction with an interactive application displayed on adisplay to provide various interactive functions. The exemplaryembodiments described herein are provided by way of example only, andnot by way of limitation.

In one embodiment, clients and/or client devices, as referred to herein,may include head mounted displays (HMDs), terminals, personal computers,game consoles, tablet computers, telephones, set-top boxes, kiosks,wireless devices, digital pads, stand-alone devices, handheld gameplaying devices, and/or the like. Typically, clients are configured toreceive encoded video streams, decode the video streams, and present theresulting video to a user, e.g., a player of a game. The processes ofreceiving encoded video streams and/or decoding the video streamstypically includes storing individual video frames in a receive bufferof the client. The video streams may be presented to the user on adisplay integral to client or on a separate device such as a monitor ortelevision.

Clients are optionally configured to support more than one game player.For example, a game console may be configured to support two, three,four or more simultaneous players (e.g., P1, P2, . . . Pn). Each ofthese players may receive or share a video stream, or a single videostream may include regions of a frame generated specifically for eachplayer, e.g., generated based on each player's point of view. Any numberof clients can be local (e.g., co-located) or are geographicallydispersed. The number of clients included in a game system may varywidely from one or two to thousands, tens of thousands, or more. As usedherein, the term “game player” is used to refer to a person that plays agame and the term “game playing device” is used to refer to a deviceused to play a game. In some embodiments, the game playing device mayrefer to a plurality of computing devices that cooperate to deliver agame experience to the user.

For example, a game console and an HMD may cooperate with the videoserver system to deliver a game viewed through the HMD. In oneembodiment, the game console receives the video stream from the videoserver system and the game console forwards the video stream, or updatesto the video stream, to the HMD and/or television for rendering.

Still further, the HMD can be used for viewing and/or interacting withany type of content produced or used, such video game content, moviecontent, video clip content, web content, advertisement content, contestcontent, gamboling game content, conference call/meeting content, socialmedia content (e.g., posting, messages, media streams, friend eventsand/or game play), video portions and/or audio content, and content madefor consumption from sources over the internet via browsers andapplications and any type of streaming content. Of course, the foregoinglisting of content is not limiting, as any type of content can berendered so long as it can be viewed in the HMD or rendered to a screenor screen of the HMD.

Clients may, but are not required to, further include systems configuredfor modifying received video. For example, a client may be configured toperform further rendering, to overlay one video image on another videoimage, to crop a video image, and/or the like. For example, clients maybe configured to receive various types of video frames, such asI-frames, P-frames and B-frames, and to process these frames into imagesfor display to a user. In some embodiments, a member of clients isconfigured to perform further rendering, shading, conversion to 3-D,conversion to 2D, distortion removal, sizing, or like operations on thevideo stream. A member of clients is optionally configured to receivemore than one audio or video stream.

Input devices of clients may include, for example, a one-hand gamecontroller, a two-hand game controller, a gesture recognition system, agaze recognition system, a voice recognition system, a keyboard, ajoystick, a pointing device, a force feedback device, a motion and/orlocation sensing device, a mouse, a touch screen, a neural interface, acamera, input devices yet to be developed, and/or the like.

A video source may include rendering logic, e.g., hardware, firmware,and/or software stored on a computer readable medium such as storage.This rendering logic is configured to create video frames of the videostream based on the game state. All or part of the rendering logic isoptionally disposed within one or more graphics processing unit (GPU).Rendering logic typically includes processing stages configured fordetermining the three-dimensional spatial relationships between objectsand/or for applying appropriate textures, etc., based on the game stateand viewpoint. The rendering logic can produce raw video that isencoded. For example, the raw video may be encoded according to an AdobeFlash® standard, HTML-5, .wav, H.264, H.263, On2, VP6, VC-1, WMA,Huffyuv, Lagarith, MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, orthe like. The encoding process produces a video stream that isoptionally packaged for delivery to a decoder on a device. The videostream is characterized by a frame size and a frame rate. Typical framesizes include 800×600, 1280×720 (e.g., 720p), 1024×768, 1080p, althoughany other frame sizes may be used. The frame rate is the number of videoframes per second. A video stream may include different types of videoframes. For example, the H.264 standard includes a “P” frame and a “I”frame. I-frames include information to refresh all macro blocks/pixelson a display device, while P-frames include information to refresh asubset thereof. P-frames are typically smaller in data size than areI-frames. As used herein the term “frame size” is meant to refer to anumber of pixels within a frame. The term “frame data size” is used torefer to a number of bytes required to store the frame.

In some embodiments, the client can be a general purpose computer, aspecial purpose computer, a gaming console, a personal computer, alaptop computer, a tablet computer, a mobile computing device, aportable gaming device, a cellular phone, a set-top box, a streamingmedia interface/device, a smart television or networked display, or anyother computing device capable of being configured to fulfill thefunctionality of a client as defined herein. In one embodiment, a cloudgaming server is configured to detect the type of client device which isbeing utilized by the user, and provide a cloud-gaming experienceappropriate to the user's client device. For example, image settings,audio settings and other types of settings may be optimized for theuser's client device.

FIG. 12 illustrates an embodiment of an Information Service Providerarchitecture. Information Service Providers (ISP) 1070 delivers amultitude of information services to users 1082 geographically dispersedand connected via network 1086. An ISP can deliver just one type ofservice, such as stock price updates, or a variety of services such asbroadcast media, news, sports, gaming, etc. Additionally, the servicesoffered by each ISP are dynamic, that is, services can be added or takenaway at any point in time. Thus, the ISP providing a particular type ofservice to a particular individual can change over time. For example, auser may be served by an ISP in near proximity to the user while theuser is in her home town, and the user may be served by a different ISPwhen the user travels to a different city. The home-town ISP willtransfer the required information and data to the new ISP, such that theuser information “follows” the user to the new city making the datacloser to the user and easier to access. In another embodiment, amaster-server relationship may be established between a master ISP,which manages the information for the user, and a server ISP thatinterfaces directly with the user under control from the master ISP. Inanother embodiment, the data is transferred from one ISP to another ISPas the client moves around the world to make the ISP in better positionto service the user be the one that delivers these services.

ISP 1070 includes Application Service Provider (ASP) 1072, whichprovides computer-based services to customers over a network. Softwareoffered using an ASP model is also sometimes called on-demand softwareor software as a service (SaaS). A simple form of providing access to aparticular application program (such as customer relationshipmanagement) is by using a standard protocol such as HTTP. Theapplication software resides on the vendor's system and is accessed byusers through a web browser using HTML, by special purpose clientsoftware provided by the vendor, or other remote interface such as athin client.

Services delivered over a wide geographical area often use cloudcomputing. Cloud computing is a style of computing in which dynamicallyscalable and often virtualized resources are provided as a service overthe Internet. Users do not need to be an expert in the technologyinfrastructure in the “cloud” that supports them. Cloud computing can bedivided in different services, such as Infrastructure as a Service(IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).Cloud computing services often provide common business applicationsonline that are accessed from a web browser, while the software and dataare stored on the servers. The term cloud is used as a metaphor for theInternet (e.g., using servers, storage and logic), based on how theInternet is depicted in computer network diagrams and is an abstractionfor the complex infrastructure it conceals.

Further, ISP 1070 includes a Game Processing Server (GPS) 1074 which isused by game clients to play single and multiplayer video games. Mostvideo games played over the Internet operate via a connection to a gameserver. Typically, games use a dedicated server application thatcollects data from players and distributes it to other players. This ismore efficient and effective than a peer-to-peer arrangement, but itrequires a separate server to host the server application. In anotherembodiment, the GPS establishes communication between the players andtheir respective game-playing devices exchange information withoutrelying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are the preferred method of hosting game servers for mostPC-based multiplayer games. Massively multiplayer online games run ondedicated servers usually hosted by the software company that owns thegame title, allowing them to control and update content.

Broadcast Processing Server (BPS) 1076 distributes audio or videosignals to an audience. Broadcasting to a very narrow range of audienceis sometimes called narrowcasting. The final leg of broadcastdistribution is how the signal gets to the listener or viewer, and itmay come over the air as with a radio station or TV station to anantenna and receiver, or may come through cable TV or cable radio (or“wireless cable”) via the station or directly from a network. TheInternet may also bring either radio or TV to the recipient, especiallywith multicasting allowing the signal and bandwidth to be shared.Historically, broadcasts have been delimited by a geographic region,such as national broadcasts or regional broadcast. However, with theproliferation of fast internet, broadcasts are not defined bygeographies as the content can reach almost any country in the world.

Storage Service Provider (SSP) 1078 provides computer storage space andrelated management services. SSPs also offer periodic backup andarchiving. By offering storage as a service, users can order morestorage as required. Another major advantage is that SSPs include backupservices and users will not lose all their data if their computers' harddrives fail. Further, a plurality of SSPs can have total or partialcopies of the user data, allowing users to access data in an efficientway independently of where the user is located or the device being usedto access the data. For example, a user can access personal files in thehome computer, as well as in a mobile phone while the user is on themove.

Communications Provider 1080 provides connectivity to the users. Onekind of Communications Provider is an Internet Service Provider (ISP)which offers access to the Internet. The ISP connects its customersusing a data transmission technology appropriate for delivering InternetProtocol datagrams, such as dial-up, DSL, cable modem, fiber, wirelessor dedicated high-speed interconnects. The Communications Provider canalso provide messaging services, such as e-mail, instant messaging, andSMS texting. Another type of Communications Provider is the NetworkService provider (NSP) which sells bandwidth or network access byproviding direct backbone access to the Internet. Network serviceproviders may consist of telecommunications companies, data carriers,wireless communications providers, Internet service providers, cabletelevision operators offering high-speed Internet access, etc.

Data Exchange 1088 interconnects the several modules inside ISP 1070 andconnects these modules to users 1082 via network 1086. Data Exchange1088 can cover a small area where all the modules of ISP 1070 are inclose proximity, or can cover a large geographic area when the differentmodules are geographically dispersed. For example, Data Exchange 1088can include a fast Gigabit Ethernet (or faster) within a cabinet of adata center, or an intercontinental virtual area network (VLAN).

Users 1082 access the remote services with client device 1084, whichincludes at least a CPU, a display and I/O. The client device can be aPC, a mobile phone, a netbook, tablet, gaming system, a PDA, etc. In oneembodiment, ISP 1070 recognizes the type of device used by the clientand adjusts the communication method employed. In other cases, clientdevices use a standard communications method, such as html, to accessISP 1070.

Embodiments of the present invention may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theinvention can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a wire-based or wireless network.

With the above embodiments in mind, it should be understood that theinvention can employ various computer-implemented operations involvingdata stored in computer systems. These operations are those requiringphysical manipulation of physical quantities. Any of the operationsdescribed herein that form part of the invention are useful machineoperations. The invention also relates to a device or an apparatus forperforming these operations. The apparatus can be specially constructedfor the required purpose, or the apparatus can be a general-purposecomputer selectively activated or configured by a computer programstored in the computer. In particular, various general-purpose machinescan be used with computer programs written in accordance with theteachings herein, or it may be more convenient to construct a morespecialized apparatus to perform the required operations.

The invention can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data, which can thereafter be read by acomputer system. Examples of the computer readable medium include harddrives, network attached storage (NAS), read-only memory, random-accessmemory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical andnon-optical data storage devices. The computer readable medium caninclude computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the claims.

The invention claimed is:
 1. A system for rendering virtual realitycontent, comprising: a housing to be attached to a head of a user via astrap; optics integrated in the housing, the optics oriented on aninternal side of the housing that is configured for orientation towardeyes of the user when the housing is attached to the head of the user; aholder integrated in the housing, the holder is configured to receive asmartphone, wherein a screen of the smartphone is oriented toward theoptics of the housing, the smartphone having a data connection with theholder to enable data communication between electronics of the housingand electronics of the smartphone; shutter control logic interfaced withthe electronics of the housing; optics control logic interfaced with theoptics; a processor configured to interface with the shutter controllogic and the optics control logic, the processor is configured to setthe shutter control logic to successively switch between rendering imagedata obtained from the screen of the smartphone to only a left eye-sideof the optics and then only to a right eye-side of the optics, theprocessor is further configured to interface with the optics controllogic so that rendering is optimized for when the image data is providedto the left and right eye-sides of the optics, wherein the shuttercontrol logic enables display of three-dimensional images from the imagedata rendered on the screen of the smartphone when viewed through theoptics of the housing; and an inertial sensor of the housing, theinertial sensor providing movement and orientation data of the housing,the movement and orientation data of the housing being communicated tothe smartphone; wherein the smartphone uses the movement and orientationdata of the housing along with movement and orientation produced by aninertial sensor of the smartphone to fine tune movement and orientationof the housing to influence or update a view direction and perspectiveinto a virtual reality scene rendered by the smartphone.
 2. The systemof claim 1, further comprising, an device interface for connecting thesmartphone to the housing.
 3. The system of claim 1, further comprising,combining at least some data from the inertial sensor of the housingwith data from the inertial sensor of the smartphone to improve trackingof the housing.
 4. The system of claim 1, further comprising, aplurality of light emitting diodes disposed on a surface of the housing,the light emitting diodes used for tracking position of the housingusing at least one camera.
 5. The system of claim 1, further comprising,a plurality of photo diodes disposed on a surface of the housing, thephoto diodes used for tracking position of the housing using at leastone light emitting device positioned in space where the housing is to betracked.
 6. The system of claim 1, further comprising, circuitry of theelectronics of the housing for processing a see-through mode, wherein afront facing camera of the smartphone is activated; wherein the shuttercontrol logic is paused and the optics control logic enable display ofimage data from the screen of the smartphone that shows a real-worldview via the front facing camera of the smartphone, when in thesee-through mode.
 7. The system of claim 1, further comprising,circuitry of the electronics of the housing for processing a see-throughmode, wherein a front facing camera of the housing is activated; whereinthe shutter control logic is paused and the optics control logic enabledisplay of image data from the screen of the smartphone that shows areal-world view via the front facing camera of the housing, when in thesee-through mode.
 8. The system of claim 1, further comprising, anetwork interface integrated with the housing, the network interfaceproviding access to a network for connecting to an internet site havingdigital content, the digital content being one of streaming content orinteractive streaming content, wherein the streaming interactive contentis to be rendered by the smartphone for display via the optics of thehousing.
 9. The system of claim 1, further comprising, a networkinterface provided by the smartphone, the network interface providingaccess to the network for connecting to an internet site having digitalcontent, the digital content being one of streaming content orinteractive streaming content, wherein the streaming interactive contentis to be rendered by the smartphone for display via the optics of thehousing.
 10. The system of claim 1, wherein the optics integrated intothe housing include a left eye optic and a right eye optic.
 11. Thesystem of claim 10, wherein user input is via a controller, a handgesture, a touch gesture, or voice control, or a combination thereof.12. The system of claim 1, wherein the housing includes a device input,the device input configured for receiving user input for controlling atleast part of an interactive content scene rendered in the image datagenerated by the smartphone.
 13. A head mounted display (HMD) system,comprising, a housing; optics disposed in the housing; an inertialsensor associated with the housing; a processor disposed in the housing;shutter control logic interfaced with the processor, the shutter controllogic configured to operate to alternate between opening a left eye-sideof the optics and closing a right eye-side of the optics to opening theright eye-side of the optics and closing the left eye-side of theoptics; and a holder portion integrated with the housing, the holderportion configured to receive a smart device having a display screen,the display screen being oriented toward the optics of the housing in adirection that is toward a face of a user when the HMD system is worn bythe user, the smart device including an inertial sensor; wherein theshutter control logic in operation is configured to enable athree-dimensional view of image data rendered on the display screen ofthe smart device; wherein data from the inertial sensor of the housingis provided to the smart device to enable fine tuning of at least onedetermination of position and orientation that is determined by thesmart device using data from the inertial sensor of the smart device,the fine tuning is used to influence or update a view into a virtualreality scene when rendered by the smart device.
 14. The head mounteddisplay (HMD) system of claim 13, wherein the shutter control logicacting to close the left eye-side or right eye-side optics is configuredto cause the optics to be opaque as opposed to transparent.
 15. The headmounted display (HMD) system of claim 13, further comprising, opticscontrol for adjusting a focus of the image data presented through theoptics.
 16. The head mounted display (HMD) system of claim 15, whereinthe adjusting includes performing image distortion adjustments, sizingadjustments, or a combination thereof.
 17. The head mounted display(HMD) system of claim 13, wherein the housing includes at least onefront facing camera.
 18. The head mounted display (HMD) system of claim13, wherein the housing includes photo diodes to enable tracking of thehousing, the photo diodes being arranged on a surface of the housing atpre-defined separations.
 19. The head mounted display (HMD) system ofclaim 13, wherein the housing includes light emitting diodes to enabletracking of the housing by a camera.
 20. The head mounted display (HMD)system of claim 13, further comprising, a network interface integratedwith the housing, the network interface providing access to the networkfor connecting to an internet site having digital content, the digitalcontent being one of streaming content or interactive streaming content,wherein the streaming interactive content is to be rendered by the smartdevice for display via the optics of the housing.
 21. The head mounteddisplay (HMD) system of claim 13, further comprising, a networkinterface provided by the smart device, the network interface providingaccess to the network for connecting to an internet site having digitalcontent, the digital content being one of streaming content orinteractive streaming content, wherein the streaming interactive contentis to be rendered by the smart device for display via the optics of thehousing.